Papers with multimodal information extraction
Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent multimodal information extraction approaches overestimate the significance of images. |
| Approach: | They propose a general data splitting strategy to divide social media posts into two sets to achieve better performance under information extraction models of the corresponding modalities. |
| Outcome: | The proposed method outperforms existing models on two different multimodal information extraction tasks. |
MAKAR: a Multi-Agent framework based Knowledge-Augmented Reasoning for Grounded Multimodal Named Entity Recognition (2025.emnlp-main)
Copied to clipboard
Xinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang, Hongzhang Mu, Yubin Wang, Gaopeng Gou, Li Qian, Li Peng, Wei Liu, Jian Luan, Hongbo Xu
| Challenge: | Existing methods for GMNER fail to address semantic ambiguity caused by polysemy and long-tail distribution of datasets. |
| Approach: | They propose a framework for Grounded Multimodal Named Entity Recognition that leverages a Multimodal Large Language Model to address semantic ambiguity. |
| Outcome: | Extensive experiments show that the proposed framework outperforms existing methods on two benchmark datasets. |
Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models train a visual encoder with weak cross-modal supervision signals, resulting in a limited capacity to capture non-textual features and suboptimal performance. |
| Approach: | They propose a Visually-Asymmetric coNsistenCy Learning approach that enhances the model’s ability to capture fine-grained visual and layout features through the incorporation of color priors. |
| Outcome: | The proposed approach outperforms the strong LayoutLM series baseline on benchmark datasets and provides insights for optimizing model performance. |